[AMD] dsv4-fp4-mi355x-atom: enable DPA TBO at high concurrency, update image to atom0.1.4 by seungrokj · Pull Request #1717 · SemiAnalysisAI/InferenceX

seungrokj · 2026-06-12T12:41:18Z

Summary

Update image to rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612
Enable --enable-tbo (Token-Bucket Overlap) on top of DPA+TP8 at high concurrency:
- ISL=1024 / OSL=1024 → TBO at CONC ≥ 1024
- ISL=8192 / OSL=1024 → TBO at CONC ≥ 256
Update ISL=8192 search-space: TP8-only conc=4–64, DPA conc=128–1024 (previously conc=1–64 and DPA conc=64–512)
Add perf-changelog entry

Motivation

Based on Pareto frontier analysis from ATOM runs 27367309656 and 27030375093, DPA TBO dominates the Pareto frontier at high concurrency for both ISL settings, delivering the best throughput/GPU without sacrificing interactivity vs plain DPA.

Test plan

Verify bash -n syntax check passes on dsv4_fp4_mi355x_atom.sh
Confirm TBO is enabled for correct ISL/CONC combinations in CI run

🤖 Generated with Claude Code

Note

Medium Risk
Changes ATOM launch flags (TBO, cudagraph capture, prefix caching, max concurrent seqs) for a large production model benchmark path; the CI sweep is temporarily collapsed to a single concurrency point, which limits regression coverage until the commented grid is re-enabled.

Overview
Updates DeepSeek-V4 FP4 on MI355X (ATOM) benchmarking to match Pareto-tuned high-concurrency serving and a newer ROCm ATOM image.

CI config (dsv4-fp4-mi355x-atom): Bumps the container to atom0.1.4_20260612, removes the prior Day-0 CONC=1 guard comment, and replaces the active fixed-seq-len sweep with a narrow setup—the full TP8 / DPA / DPA+TBO grids are left commented, and the only live point is ISL=8192, DPA, CONC=1024.

Benchmark script (dsv4_fp4_mi355x_atom.sh): On DPA+TP8, turns on --enable-tbo when ISL/OSl/concurrency cross the thresholds (1k/1k at CONC≥1024; 8k/1k at CONC≥256), with expanded --cudagraph-capture-sizes for the 1k/1k TBO case. The ATOM server now gets --no-enable_prefix_caching, --cudagraph-capture-sizes, and --max-num-seqs ${CONC}; eval-only runs wire context length via compute_eval_context_length (with --max-model-len left commented).

Documents the change in perf-changelog.yaml.

^{Reviewed by Cursor Bugbot for commit 7ffa976. Bugbot is set up for automated code reviews on this repo. Configure here.}

…e image to atom0.1.4 - Enable --enable-tbo for ISL=1024/OSL=1024 at CONC>=1024 and ISL=8192/OSL=1024 at CONC>=256 - Update image to atom0.1.4_20260612 - Update ISL=8192 search-space to start at conc=4 and use DPA from conc=128 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T12:59:36Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27416363770
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27416363770

…onc range - Pass --max-model-len to server using SERVE_MAX_MODEL_LEN - Add EVAL_ONLY path: compute eval context length via compute_eval_context_length - Extend conc-end to 8192 (isl=1024) and 4096 (isl=8192) in amd-master.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T13:18:19Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27417198968
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27417198968

github-actions · 2026-06-12T14:23:22Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

github-actions · 2026-06-12T17:08:40Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

…sable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor · 2026-06-13T13:08:52Z

+        # conc4-64, TP8
+        # conc128, DPA
+        # conc256-4096, DPA TBO
+      - { tp: 8, ep: 1, dp-attn: true, conc-start: 1024, conc-end: 1024 }


Search space not enabled

High Severity

The dsv4-fp4-mi355x-atom fixed-seq-len matrix no longer matches the PR: the ISL=1024 scenario is fully commented out, and ISL=8192 only runs a single DPA point at conc=1024 instead of the documented TP8 (conc 4–64) and DPA (conc 128–1024) sweeps.

^{Reviewed by Cursor Bugbot for commit c3b3289. Configure here.}

cursor · 2026-06-13T13:08:52Z

    --gpu-memory-utilization 0.85 \
+    --no-enable_prefix_caching \
    > $SERVER_LOG 2>&1 &
+    #--max-model-len "$SERVE_MAX_MODEL_LEN" \


Eval context not wired

Medium Severity

This commit adds SERVE_MAX_MODEL_LEN / compute_eval_context_length handling for EVAL_ONLY, but the server launch leaves --max-model-len commented out, so the ATOM server never receives the computed context length.

^{Reviewed by Cursor Bugbot for commit c3b3289. Configure here.}

…m-seqs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7ffa976. Configure here.}

cursor · 2026-06-13T13:11:54Z

+            CUDAGRAPH_SIZES='[1,2,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140,144,148,152,156,160,164,168,172,176,180,184,188,192,196,200,204,208,212,216,220,224,228,232,236,240,244,248,252,256,512,1024]'
+        elif [ "$ISL" -eq 8192 ] && [ "$OSL" -eq 1024 ] && [ "$CONC" -ge 256 ]; then
+            PARALLEL_ARGS=(-tp "$TP" --enable-dp-attention --enable-tbo)
+        else


CUDA graph sizes omit high concurrency

Medium Severity

For ISL=8192 and OSL=1024, --enable-tbo turns on when CONC≥256, but CUDAGRAPH_SIZES stays capped at 256 while the server uses --max-num-seqs equal to CONC (e.g. 1024 in the active YAML). The 1024/1024 TBO branch extends capture sizes through 1024; this path does not.

^{Reviewed by Cursor Bugbot for commit 7ffa976. Configure here.}

functionstackx · 2026-06-13T16:20:20Z

can u rebase it so the sweep triggers?

seungrokj and others added 2 commits June 12, 2026 21:38

[AMD] perf-changelog: dsv4-fp4-mi355x-atom DPA TBO + image atom0.1.4

c566e28

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj requested a review from a team June 12, 2026 12:41

seungrokj requested review from 1am9trash, billishyahao, chunfangamd and yctseng0211 as code owners June 12, 2026 12:41

github-project-automation Bot added this to InferenceMAX Board Jun 12, 2026

[AMD] perf-changelog: add PR link #1717

7e1aa06

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj added AMD full-sweep-enabled labels Jun 12, 2026

[AMD] dsv4_fp4_mi355x_atom.sh: disable prefix caching

65e0fa3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj removed the full-sweep-enabled label Jun 13, 2026

[AMD] dsv4-fp4-mi355x-atom: narrow eval to single conc=1024 point, di…

c3b3289

…sable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 13, 2026

View reviewed changes

[AMD] dsv4_fp4_mi355x_atom.sh: add cudagraph-capture-sizes and max-nu…

7ffa976

…m-seqs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMD] dsv4-fp4-mi355x-atom: enable DPA TBO at high concurrency, update image to atom0.1.4#1717

[AMD] dsv4-fp4-mi355x-atom: enable DPA TBO at high concurrency, update image to atom0.1.4#1717
seungrokj wants to merge 7 commits into
mainfrom
amd/dsv4_atom_0612

seungrokj commented Jun 12, 2026 •

edited by cursor Bot

Loading

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

cursor Bot Jun 13, 2026

Uh oh!

cursor Bot Jun 13, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Jun 13, 2026

Uh oh!

functionstackx commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

seungrokj commented Jun 12, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Test plan

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

cursor Bot Jun 13, 2026

Choose a reason for hiding this comment

Search space not enabled

Uh oh!

cursor Bot Jun 13, 2026

Choose a reason for hiding this comment

Eval context not wired

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 13, 2026

Choose a reason for hiding this comment

CUDA graph sizes omit high concurrency

Uh oh!

functionstackx commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

seungrokj commented Jun 12, 2026 •

edited by cursor Bot

Loading